Research Brief 
Center for the Advancement of 
Engineering Education 
NSF Grant ESI-0227558 



CENTER FOR THE ADVANCEMENT OF ENGINEERING EDUCATION 


Research Design Becomes Research Reality: Colorado 
School of Mines Implements Research Methodology for the 
Center for the Advancement of Engineering Education 

Author: Heidi Loshbaugh, Ruth Streveler, Kimberley Breaux 
Source: 2005 American Society for Engineering Education Conference 

The Center for the Advancement of Engineering Education was founded in 2003 with five 
collaborating institutions. A multi-institutional, multi-year grant offers many opportunities for 
the demands of reality to interfere with design goals. In particular, at Colorado School of Mines 
(CSM) student demographics required adjustment of the original APS research design. The 
following paper describes the challenges and solutions of the recruitment process at CSM. 

Method and Background 

The Academic Pathways Study (APS) element of the CAEE involves surveys, structured 
interviews, ethnographic interviews, and a performance task. The following describes the 
campus-specific implementation of this collaborative study and how research design 
implementation must conform to the reality of a specific target campus and its representative 
population. 

In the first year of the APS, primary tasks were selecting 
the participant pool and collecting data through 
ethnographic observations, on-line surveys, and 
interviews. Most CAEE communication takes place by e- 
mail and telephone conferences due to the great distances 
between campuses and the large number of researchers 
involved. In the first year, conference calls were used to 
clarify, refine, and implement the research, and two face- 
to-face meetings of research team members were held for making decisions about research 
design and policy. 

In the first year of the study (academic year 2003-2004), CSM had 750 incoming, first-year 
students. To be eligible for the APS study, CSM participants had to be May/June 2003 high 
school graduates, eighteen or older by October 1, 2003, and a US citizen or permanent resident. 
CSM also required participants to have enrolled in or have the intention to enroll in an ABET- 
accredited engineering major (3 CSM majors were excluded: chemistry, math and computer 
science, and economics and business). 

The CSM sample design was fifty percent female, fifty percent male for both study and control 
groups. Fifteen participants would be White/Asian, five would be Latino/a, Native American, or 
African American. CSM’s total APS population would be eighty students: forty enrolled in the 
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control group, thirty two participating in interviews and surveys, and 8 participating in all APS 
methods. The control group was designed to verify whether or not participation in APS would 
affect the success of study/ethnographic participants. In year two, the control groups were 
disbanded at all institutions in recognition of the impossibility that participation would have no 
effect, given the frequent contact participants have with the researchers. 

What We Found 

CSM's size presented additional challenges to fulfillment of the study design. First, there were 
personnel constraints since CAEE was integrated into CSM's one-person Center for Engineering 
Education (CEE) at CSM. Second, the primary researcher left temporarily for maternity leave in 
the first year and was replaced by two other researchers. Third, REU students were employed as 
part of the multi-method team. However, CSM's small size required careful attention to detail to 
protect the confidentiality of APS participants who might share classrooms, living 
accommodations, or dining facilities with undergraduate employees of CAEE. 

Participants were recruited at nine events in the 2003-2004 academic year. Researchers received 
342 signatures indicating willingness to participate - 98 were women and 244 were men. Forty- 
four signatures were from ethnic minorities. These students were invited to information sessions 
where interested and eligible students signed consent forms. After these sessions, the team still 
lacked enough females to complete the sampling plan and had no African-American candidates. 
Two more recruiting sessions for female students were scheduled and the primary researcher 
attended a meeting of the National Society of Black Engineers (NSBE) in an attempt to recruit 
African Americans. Ultimately, the team chose not to pursue these students further and were thus 
unable to include any African-American students in the study sample. The team also had to 
revise downward the sample of women. 

At the end of the recruiting sessions the team had received 112 consent signatures. Forty of these 
were women, one from an ineligible major. CSM lacked enough non-Caucasian females to meet 
its designated sample and instead over-sampled for ethnic-minority males (Latinos and Asian 
American/Pacific-Islanders). At CSM, broad representation is a relative term as it has about 77 
percent in-state enrollment, 77 percent males, and 86 percent Whites. The team placed 
participants according to majors, hometowns, ethnicity, sex, and the answers to questions on a 
recruiting questionnaire. Students who responded “money” as their primary motivation or who 
selected "academic records only” as their principal level of interest on the questionnaire were 
eliminated. 

Furthermore, there were ongoing challenges associated with scheduling interviews and 
observations that resulted in time pressures. Researchers needed to be mindful of the 
participants’ value however, remaining accommodating and tactful even when participants 
required multiple prompts to complete tasks. Human subjects approval was also a challenge as 
CSM lacks an Institutional Review Board (IRB). Howard University sponsored CSM’s informed 
consent protocols. As the APS methodology was refined and updated, the informed consent 
protocols also required updating and new signatures from participants. 

Because of the need to have protocols in place before data collection could begin, and the 
research imperative to collect data at the same time on all the campuses, CSM could not begin 
data collection before subjects were in place at the collaborating institutions. Thus, students at 
CSM were forced to wait several weeks before their participation actually began. Variations 
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among institutional calendars can hinder the effectiveness of cross-institutional research. 
Anticipating and planning for this complication could make such multiple-institution research 
run more smoothly. 

Implications of Findings 

To enact fundamental change in engineering education, cross-campus and multi-method research 
like CAEE will be ever more common. Managing the transition from research design into 
practice is critical for success. Anticipating scheduling challenges and coordination of different 
academic calendars is critical, as is maintaining team-oriented communication alongside task- 
based functions. Flexibility is important as research design changes to meet the pragmatic 
realities of each campus. 
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